Do we still use caching to reconstruct the prior conversation using the local DB, or are we using only the service side records?

Only the service-side records. There is no local database read on this path.

Concretely: reconstructHistoryIfNeeded calls fetchSessionRecords, which POSTs to /sessions/records/query on the API and folds the returned rows into ChatMessage[]. Nothing in that path consults sandbox-local state.

There is a separate mechanism that can look like caching, and it is worth keeping distinct. When a sandbox is still parked from the previous turn, the harness process inside it already holds its own conversation context, so the runner sends only the new text instead of replaying anything. That is the warm path, and it never touches records.

The two interact badly today, which is the part I would not have predicted from the design. With last-message-only on, the warm path is rejected on every turn, because the warm check fingerprints the inbound history and the inbound history is now a single message. So in practice you never stay warm, and every turn falls through to the records path. Evidence is in finding 2 of the QA comment.

My question was related to the playground and not the runner.

Only server-side records. The runner fetches the session's durable record log (records-query.ts) and folds it; no local/IDB cache is involved in reconstruction.

This is not an accurate description. Just making sure that the design and the implementation take that into account. There is warm, there is cold, there is cold evicted, and I think the issue happens only when cold evicted and we need to rebuild the whole transcript again.

You are right, and the QA shows the implementation carries the same gap, not just the write-up.

The three states as they actually exist in the runner:

• Warm. A sandbox from the previous turn is still parked and is accepted for reuse. The harness process inside it still holds the conversation, so the runner sends only the new text. Nothing is replayed.
• Cold. No parked sandbox exists. The runner creates one and replays the whole transcript into a fresh harness process.
• Cold evicted. A parked sandbox exists but fails a compatibility check, so it is destroyed and the turn degrades to cold. The check is at services/runner/src/server.ts:589-609, and it rejects for one of four reasons: config, history, credentials, or tail.

Your instinct is right that the interesting case is the evicted one, and it is worse than the design assumes. The history reason compares a fingerprint of the inbound prior conversation against what the previous turn stored. With last-message-only, the inbound prior conversation is empty on every fresh turn, so that comparison can never match. Every multi-turn conversation is therefore evicted to cold on every turn after the first.

Measured on the three-turn journey: turn 2 went from 1570 ms to 4623 ms and turn 3 from 1399 ms to 4252 ms, with three mismatch (history) evictions in the runner log and none in the baseline. Finding 2 in the QA comment.

I will rewrite this section of the design doc against the three real states as part of the fixes.

Agreed, the doc conflated them. Corrected in 4c34ce4: records-based reconstruction supplies model context only in the cold-evicted case; warm and cold-not-evicted resume natively. Reconstruction still runs on warm turns, but only to realign the keep-alive fingerprint.

Did we do a live queue here where we compared the reconstructed session that you get from the records in the runner (like this logic) to the stuff that is sent by the frontend in different scenarios, like:

• tool calls
• warm approvals
• cold approvals
• MCPs
• requests
• questions
• long discussions
  because I think that's the real validation that we can remove this flag?

Please spin off this in a work tree, and let's use the Claude Code subscription with Hi-Co and do a QA using this. FDPR obviously doesn't say that it did it before.

No, that comparison had not been done. It has now, and you were right that it is the validation that matters: it found four defects that the pass/fail suite cannot see.

Method: one dev stack at this PR's head with all four flags on, then the agent release gate run twice against it, changing only how the client sends history. Run A sends the full conversation, which is today's frontend. Run B sends only the trailing user message, mirroring agentRequest.ts with the flag on. Instrumentation in the runner dumps the post-reconstruction request.messages, so the two runs can be diffed on the exact array the model receives.

Both runs are all-PASS on all nine journeys, which is the whole point. The differences only show in the message arrays and the runner log.

What it found:

1. Reconstruction replays the current prompt twice on every first turn. This fires even with the frontend flag off, because turn one always carries exactly one message and the inbound.length > 1 guard does not block it.
2. Last-message-only evicts the warm session on every turn, so every multi-turn conversation runs cold.
3. Smart truncation raises AttributeError and drops the oversized record instead of truncating it.
4. The dropped-record counter has no consumer outside the tests.

One scenario from your list is still untested: a fresh turn after an approval resume, which is where Codex predicted a duplicated tool-call id. The gate's approve journey stops at the resume, so it never exercised it.

Full detail and evidence: QA comment.

Done. Worktree at fix/sessions-reconstruct-qa, branched from this PR's head, with its own deployment so it does not collide with the other stacks on the box. The agent ran on the mounted Claude Code subscription with haiku.

One thing had to be fixed before the QA could run at all: the runner service deliberately has no env_file, and AGENTA_SESSIONS_RECONSTRUCT and AGENTA_RECORDS_DURABLE were not listed in any of the four compose files. So the feature could not be switched on in any deployment, only in unit tests. That is now wired in all four editions and documented in the env examples, as the first commit on the fix branch.

Results are in the QA comment. Fixes will land on that branch as a PR stacked on this one, so the fix diff stays readable on its own.

Yes — that differential QA ran (see the QA comment on this PR). It diffed reconstructed vs FE-sent history across tool calls, approvals, MCP, and multi-turn, and surfaced 4 defects a pass/fail gate can't see. Fixes are in the stacked PR #5500. The flag stays until a clean re-run.

Done — the QA above ran that way (worktree at this PR's head, Claude subscription / Hi-Co, all four flags on). Findings + fixes: #5500.

What is the behavior when this fails?

Today it fails silently, and with last-message-only on that means the agent forgets the conversation.

fetchSessionRecords catches every error, logs one line to stderr, and returns null. The caller in reconstruct-history.ts treats null as "leave the inbound history alone". The doc comment describes that as falling back to the inbound history "rather than run with an empty context", which is true only while the client still sends the full conversation.

Once the client sends one message, the inbound history is one message. So a failed query means the runner hands the model a single user turn with no prior context. The model answers as if the user had just arrived. There is no error frame, no retry, and nothing on the wire that tells the user or the caller that this happened. The only trace is one stderr line in the runner.

Two related gaps in the same function:

• There is no timeout. Node's global fetch applies no per-request deadline; the underlying defaults are 300 seconds for headers and body. A hung API stalls the turn rather than degrading it.
• An empty result and a failed result are treated the same by the caller. [] and null both end in "keep the inbound history", so a genuinely empty log and an unreachable API are indistinguishable downstream.

My recommendation for the fix: when the client has already dropped its history, a failed read is not recoverable, so the turn should fail loudly instead of answering with no context. The alternative, and it is the more robust shape, is to not let the client drop history until the runner has confirmed the record log is readable and complete. That also addresses the dropped-record counter having no consumer, which is finding 4 in the QA comment.

Returns null; the caller leaves the inbound history untouched (falls back to whatever the client sent), so a failed fetch degrades to today's behavior rather than an empty context. Now also bounded by a 5 s AbortSignal.timeout (#5500) so a stall fails fast instead of hanging the turn.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

#!/bin/bash
# Verify the parent Agenta config class exposes a `sessions` field wired to SessionsConfig.
rg -n -B5 -A5 'sessions\s*:\s*SessionsConfig' api/oss/src/utils/env.py
rg -n 'class AgentaConfig' -A 30 api/oss/src/utils/env.py

#!/bin/bash
set -euo pipefail

echo "== env.py config definitions/references =="
rg -n 'SessionsConfig|SessionsRecordsConfig|utora|agenta|AGENTA_SESSIONS|SESSIONS' api/oss/src/utils/env.py

echo
echo "== relevant env.py sections =="
sed -n '1,120p' api/oss/src/utils/env.py
sed -n '400,560p' api/oss/src/utils/env.py

echo
echo "== all project references to SessionsConfig/SessionsRecordsConfig =="
rg -n 'SessionsConfig|SessionsRecordsConfig|agenta\.sessions|AGENTA_SESSIONS' api/oss/src || true

echo
echo "== streaming.py usage context =="
sed -n '1,180p' api/oss/src/core/sessions/records/streaming.py

Wire agenta.sessions into AmentaConfig.

env.agenta.sessions.records.smart_truncation exists in the new config structs, but AmentaConfig does not expose a sessions: SessionsConfig field, so reading this path now raises AttributeError inside the oversized-attributes try/except, logs a publish failure, and returns False; add/wire the session sub-namespace instead of silently disabling the record publish.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Require exactly one inbound user message before reconstructing.

The current inbound.length > 1 guard also reconstructs for zero messages or a single assistant message. After merging, resolvePromptText can select a historical user prompt and re-run it as the current turn. Restrict this path to one user message and add regressions for empty and assistant-only input.

 const inbound = request.messages ?? [];
-// The client already sent the conversation — nothing to rebuild.
-if (inbound.length > 1) return null;
+// Reconstruct only for a fresh turn containing exactly its trailing user message.
+if (inbound.length !== 1 || inbound[0]?.role !== "user") return null;

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

#!/bin/bash
# Confirm every path that can populate persistFailures also reliably drains it via takePersistFailures.
rg -n 'takePersistFailures' services/runner/src -g '*.ts'

#!/bin/bash
set -euo pipefail

echo "== persist.ts outline/size =="
wc -l services/runner/src/sessions/persist.ts
ast-grep outline services/runner/src/sessions/persist.ts --view expanded || true

echo
echo "== relevant persist.ts sections =="
cat -n services/runner/src/sessions/persist.ts | sed -n '1,240p'

echo
echo "== exact symbol usages repo-wide =="
rg -n 'persistFailures|takePersistFailures|durableMaxRetries|drainPersistFailures' . -g '*.ts' -g '*.tsx' -g '*.js' -g '*.jsx' -g '*.py' || true

echo
echo "== all drain / flush related symbols in runner src =="
rg -n 'drainPersist|flush|persist|reconstruct|turn-end|takePersistFailures' services/runner/src -g '*.ts' -g '*.tsx' || true

#!/bin/bash
set -euo pipefail

echo "== server.ts session-owned run wrap section =="
cat -n services/runner/src/server.ts | sed -n '930,1045p'

echo
echo "== runner src files mentioning session-specific functions =="
rg -n 'flushPersist|persistError|drainPersist|takePersistFailures|persistingEmit|buildPersistingEmitter' services/runner/src -g '*.ts'

echo
echo "== behavioral probe: exponential backoff and durable max retries =="
node - <<'JS'
const INGEST_RETRY_BASE_MS = 100;
function durations(maxRetries) {
  let total = 0;
  const times = [];
  for (let attempt = 1; attempt < maxRetries; attempt++) {
    const backoff = INGEST_RETRY_BASE_MS * 2 ** (attempt - 1);
    times.push(backoff);
    total += backoff;
  }
  return { maxRetries, totalMs: total, times };
}
for (const n of [6, 10, 15, 20]) {
  const r = durations(n);
  console.log(`${r.maxRetries}: ${r.times.length} delays, total=${r.totalMs}ms last=${r.times.at(-1)}ms`);
}
const bad = Number("0x10000000000000001"); // Number precision example; still finite by Number.isFinite
console.log("Number.maxSafeInteger:", Number.MAX_SAFE_INTEGER);
console.log("Number is finite for MaxSafeInteger:", Number.isFinite(Number.MAX_SAFE_INTEGER));
console.log("Backoff for MAX_SAFE_INTEGER:", INGEST_RETRY_BASE_MS * 2 ** (Number.MAX_SAFE_INTEGER - 1));
JS

echo
echo "== static tail summary for drain/eviction of persist failures =="
python3 - <<'PY'
from pathlib import Path
p = Path("services/runner/src/sessions/persist.ts")
s = p.read_text()
checks = {
    "takePersistFailures only deletes": 
        "set(" not in s[s.find("function takePersistFailures"):s.find("export function buildPersistingEmitter")]
        and "persistFailures.delete(sessionId)" in s[s.find("function takePersistFailures"):s.find("export function buildPersistingEmitter")],
    "no other takePersistFailures imports/usages shown in persist.ts": 
        s.count("takePersistFailures") == 1,
    "persistFailures gets populated in exhaustive durable path": 
        "persistFailures.set(sessionId" in s,
}
for k,v in checks.items():
    print(f"- {k}: {v}")
PY

Bound durable ingest retries and add a non-zero failure bound.

durableMaxRetries() reads an env override with no ceiling while the backoff doubles each attempt; a larger value delays the synchronous turn-end drain by seconds per attempt. The “can’t hang indefinitely” comment only holds for the default cap. Also, AGENTA_RECORDS_DURABLE=true can now create persistFailures entries that are never drained unless reconstruction is explicitly enabled, leaving abandoned session keys in memory.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

#!/bin/bash
set -euo pipefail

echo "== locate records-query.ts and related references =="
git ls-files | rg '(^|/)records-query\.ts$|reconstruct-history|sessions|runner' | sed -n '1,200p'

echo
echo "== records-query.ts =="
cat -n services/runner/src/sessions/records-query.ts

echo
echo "== references to fetchSessionRecords =="
rg -n "fetchSessionRecords" .

echo
echo "== package node version constraints =="
for f in package.json services/runner/package.json; do
  if [ -f "$f" ]; then echo "--- $f"; cat "$f" | sed -n '1,120p'; fi
done

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
import json, re, subprocess
import pathlib
files = [pathlib.Path(p) for p in subprocess.check_output(['git','ls-files'], text=True, encoding='utf-8').splitlines() if p.endswith(('.jsonc')))
for f in files:
    try:
        data=json.loads(f.read_text())
    except Exception:
        continue
    s=str(data)
    if 'node' in s.lower() or 'engines' in data:
        print(f"--- {f}")
        print(data.get('engines',{}))
        print(data.get('packageManager'))
        print(data.get('devEngines'))
PY

#!/bin/bash
set -euo pipefail

echo "== reconstruct-history.ts outline and relevant section =="
ast-grep outline services/runner/src/engines/sandbox_agent/reconstruct-history.ts || true
cat -n services/runner/src/engines/sandbox_agent/reconstruct-history.ts | sed -n '1,120p'

echo
echo "== Dockerfile runner Node base =="
for f in services/runner/docker/Dockerfile services/runner/docker/Dockerfile.dev services/runner/docker/Dockerfile.gh hosting/railway/oss/runner/Dockerfile; do
  if [ -f "$f" ]; then
    echo "--- $f"
    rg -n "node|FROM|pnpm|tsx" "$f" || true
  fi
done

echo
echo "== fetchSessionRecords call sites =="
cat -n services/runner/src/engines/sandbox_agent/reconstruct-history.ts | sed -n '1,80p'

Add a timeout to the session-records fetch.

fetchSessionRecords is awaited during reconstruction before prompt construction, and the runner uses Node 24 global fetch. A stalled query can therefore wait for Undici’s long request-level timeout instead of failing fast to the inbound-history fallback. Add an AbortSignal.timeout(...) on this request.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Do not treat a non-empty session ID as proof that history is reconstructable.

This enables last-message-only for sessions with no durable records. If record lookup is empty or fails, the runner preserves the now-minimal request, so prior client history is lost. Gate this on an explicit server-confirmed durable-history/readiness state; otherwise retain full history. This is required by the documented “server-known with records” fallback.

How does this work in the case of approval? What kind of message is sent? Do we always send a message from type user?

No, it is not always a user message, and that is exactly what makes the approval work.

The trim is conditional on the trailing message's role:

lastMessage?.role === "user" ? [lastMessage] : history

On an approval, the browser does not send a new user turn. It re-POSTs the whole conversation with the tool part rewritten to state: "approval-responded" and an approval: {id, approved} envelope, so the trailing message is the assistant turn carrying the settled tool result. That fails the role === "user" test, the full history is sent, and the tool result still sits next to its original tool call, which is what lets the approval bind. Same for the client-tool resume, which uses the same in-band shape.

So the intent is right. Two things I would still change.

First, the runner-side guard is not the mirror image of this rule. reconstruct-history.ts skips reconstruction when inbound.length > 1, which is a different question from "did the client send a fresh user turn". It also lets reconstruction run for zero messages and for a single assistant message. CodeRabbit flagged the same line.

Second, the two rules already disagree in production, on turn one. A first turn carries exactly one message whatever the frontend flag says, so inbound.length > 1 is false, reconstruction runs, and the prompt that was persisted moments earlier comes back and is appended to itself. That is finding 1 in the QA comment, and it reproduced on nine out of nine first turns with the frontend flag off.

The clean version is for both sides to agree on one predicate, something like "this request carries exactly its own fresh user turn", and for the runner to derive its decision from that rather than from a message count.